Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks*

Abstract Natural gradient descent (NGD) helps to accelerate the convergence of dynamics, but it requires approximations in large-scale deep neural networks because its high computational cost. Empirical studies have confirmed that some NGD methods with approximate Fisher information converge sufficiently fast practice. Nevertheless, remains unclear from theoretical perspective why and under what conditions such heuristic work well. In this work, we reveal that, specific conditions, achieves same global minima as exact NGD. We consider infinite-width limit, analyze asymptotic training dynamics function space via tangent kernel. space, are identical those information, they quickly. The holds layer-wise approximations; for instance, block diagonal approximation where each corresponds a layer well tri-diagonal K-FAC approximations. also find unit-wise assumptions. All these different an isotropic plays fundamental role achieving properties training. Thus, current study gives novel unified foundation which understand learning.